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Abstract 

Performance-Based  Logistics  (PBL)  is  an  initiative  that  the  Department  of  Defense 
(DoD)  has  targeted  for  ’’aggressive  implementation”  in  FY  2006-2009  (Wolfowitz,  2004).  It  is  an 
initiative  intended  to  improve  weapon-system  logistics  outcomes,  and  to  reduce  weapon  system 
life-cycle  costs.  Provider  evaluation  in  PBL  is  intended  to  center  on  clearly  specified  outcome 
metrics,  and  mutually-agreed  upon  goals  on  those  metrics  (DUSD-LMR,  2001) — with  the  idea 
that  the  DoD  knows  best  what  it  wants  in  terms  of  logistic  services,  but  the  vendor  may  know 
best  how  to  provide  those  services.  PBL  can  be  seen  as  an  extension  of  the  principle  of 
“commander’s  intent”  in  which  leadership  presents  goals,  but  subordinates  are  encouraged  to 
choose  methods  and  processes  (Apgar  &  Keane,  2004). 

Within  the  field  of  behavioral  decision  making,  there  is  a  substantial  literature  which 
shows  that  decision  makers  use  sub-optimal  heuristics  to  value  and  negotiate  agreements  such 
as  PBL  contracts  for  services;  these  are  subject  to  systematic  biases  in  judgment  when 
evaluating  performance  (Kahneman,  Slovic  et  al. ,  1982).  In  this  paper,  we  draw  a  connection 
between  the  intent  of  PBL  on  the  one  hand,  and  the  literature  on  biases  and  heuristics  on  the 
other.  Specifically,  we  review  the  literature  on  PBL  and  logistics  service  measurement  as  it 
relates  to:  1)  the  distinction  between  process  and  outcome  measures  and  2)  the  significance  of 
risk.  We  then  review  the  literature  on  behavioral  decision  making  and  performance  evaluation 
that  relates  to  those  same  two  topics,  and  develop  hypotheses  regarding:  1)  the  potential  impact 
of  process  measurement  on  outcome  measurement,  and  2)  the  absence  of  stated  metrics  and 
goals  for  the  variance  (risk)  of  outcomes.  We  develop  specific,  testable  hypotheses  from  this 
review  of  the  literature,  report  on  a  test  of  these  hypotheses  in  a  laboratory  experiment,  and 
discuss  the  implications  of  our  findings  in  practice. 

PBL  is  an  evolving  concept  within  the  DoD,  and  clarification  on  the  metrics  which  should 
be  used  to  assess  weapon-system  logistics  outcomes  has  been  recently  issued  which 
emphasizes  that  system-level  outcomes  such  as  operational  availability  should  be  used  to 
evaluate  PBL  candidates  and  the  performance  of  PBL  providers  (Wynne,  2004).  The  system- 
level  emphasis  of  this  clarification  is  significant  and  proper,  as  warfighting  outcomes  are  clearly 
only  impacted  by  system-level  (as  opposed  to  component-level)  performance. 

But  PBL  is  still  being  applied  at  the  component  level,  and  there  is  no  clear  guidance  (to 
our  knowledge)  on  how  to  link  component-level  variables  like  time-to-failure  to  system  level 
outcomes  like  operational  availability.  Indeed,  a  recent  Government  Accountability  Office 
(GAO)  report  found  that  most  of  the  185  PBL  contracts  they  were  able  to  identify  in  the  DoD 
were  written  at  the  component  or  subsystem  level,  and  they  suggested  that  contracting  at  the 
component  level  should  continue  to  be  preferred  to  contracting  at  the  system  level  (GAO,  2004). 

Even  in  the  private  sector,  the  measurement  and  performance  assessment  of  logistic 
services  is  known  to  be  a  difficult  task.  Proper  valuation  of  the  outcomes  of  logistic  services  (as 
opposed  to  merely  valuing  inputs,  such  as  cost)  must  include  some  assessment  of  difficulty  to 
quantify  factors  such  as  customer  satisfaction,  and  risk  reduction  (Lambert  &  Burduroglu,  2000). 
This  outcome-measurement  problem  is  made  more  difficult  because  so  many  of  the  traditional 
logistics  measures  are  process  measures  (Caplice  &  Sheffi,  1994).  Nor  is  this  situation  easier 
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when  the  services  are  provided  to  the  DoD,  yet,  outcomes  cannot  be  reduced  to  measurable 
quantities  like  profit  or  shareholder  value  (Camm,  Blickstein  et  al. ,  2004). 

The  DoD,  of  course,  has  no  simple  overarching  valuation  metric  such  as  profit,  and  it 
has  no  simple  revenue  surrogates.  Valued  outcomes  have  to  do  with  military  missions;  thus, 
even  if  logistic  services  for  a  weapon  system  are  provided  at  an  aggregate  level  by  a  single 
provider,  they  are  difficult  to  value  and  price.  At  the  level  of  a  subassembly  or  single  logistic 
element,  the  problem  is  compounded.  Unless  decision  makers  have  comprehensive  models  of 
weapon-systems  logistics,  (in  which  the  important  performance  dimensions  of  all  critical 
components  are  modeled),  they  cannot  value  a  component-level  contract  in  terms  of  system- 
level  outcomes  like  operational  availability.  Such  models  have  not  been  required,  and  we  have 
no  evidence  that  they  are  being  used  in  the  field. 

In  situations  without  clearly  observable  outcome  measures  and  valuation  functions, 
decision  makers  are  known  to  place  a  heavy  weight  on  surrogates  (such  as  process  measures 
or  even  input  measures)  (Chinander  &  Schweitzer,  2003).  Some  of  these  surrogates,  however, 
may  not  correlate  well  with  system-level  outcomes.  Under  PBL,  decision  makers  must 
determine  relevant  outcomes  for  component-level  contracts  and  separate  diagnostic  measures 
(those  that  correlate  well  with  desired  system  outcomes)  from  non-diagnostic  ones.  However, 
decision  makers  are  known  to  pursue  information  even  when  it  is  non-diagnostic  and  non¬ 
instrumental  (knowledge  of  the  measure  would  not  or  should  not  change  decisions). 
Unfortunately,  once  obtained,  such  non-instrumental  information  may  be  treated  as  if  it  were 
instrumental  (Bastardi  &  Shafir,  1998).  That  is,  decision  makers  pursue  information  they  do  not 
need,  then  act  upon  it.  In  our  paper,  we  investigate  this  tendency  in  decision  makers  who  were 
asked  to  evaluate  provider  performance  under  a  hypothetical  PBL  contract. 

It  might  be  claimed  that  additional  information  could  never  hurt  the  decision  process 
(aside  perhaps  from  the  cost  of  gathering  it),  but  at  least  two  sets  of  research  findings  indicate 
that  such  confidence  would  be  misplaced.  The  curse  of  knowledge  is  a  dysfunctional  decision¬ 
making  pattern  that  occurs  when  a  decision  maker  knows  information  they  would  be  better  off  to 
ignore;  yet,  once  it  is  known,  they  cannot  ignore  it  (Camerer,  Loewenstein  et  al.,  1989).  The 
classic  example  is  a  wine  merchant  who  over-prices  his  good  wine  and  under-prices  his  bad 
wine;  thus,  he  loses  revenue  on  both  sides  from  customers  who  do  not  know  as  much  about 
wine  as  he  does.  Thus,  the  wine  merchant  is  ’’cursed”  by  his  superior  knowledge  of  wine 
quality,  and  he  loses  revenue:  he  would  be  better  off  to  price  his  wines  according  to  market 
demand.  In  our  case,  the  decision  maker  who  pursues  non-diagnostic  process  information  may 
misestimate  provider  performance  because  of  it.  A  related  bias  is  the  dilution  effect:  the 
tendency  for  non-diagnostic  information  to  cause  diagnostic  information  to  be  undervalued 
(Nisbett,  Zukier  et  al.,  1981).  In  the  case  of  PBL,  if  a  decision  maker  captures  process  metrics, 
he  or  she  may  not  be  able  to  place  them  in  the  proper  context  relative  to  a  system-level 
outcome,  and  the  impact  of  an  important  outcome  metric  may  be  diluted.  In  our  paper,  we 
investigate  the  tendency  of  decision  makers  to  dilute  system  outcomes  when  given  knowledge 
of  process  variables. 

A  special  case  of  the  misuse  of  non-diagnostic  information  is  the  use  of  information 
about  inputs.  The  input  bias  is  the  tendency  to  make  judgments  about  the  quality  of  outcomes 
based  on  the  value  of  inputs  (Chinander  &  Schweitzer,  2003).  For  example,  people  tend  to 
judge  the  quality  of  a  product  or  service  higher  when  they  have  to  wait  longer  for  it  (Maister, 
1985).  This  bias  is  thought  to  play  an  especially  significant  role  in  evaluation  when  outcomes 
are  difficult  to  observe  or  measure.  In  the  case  of  PBL  contracts,  the  evaluation  of  proposals 
based  solely  on  the  relative  cost  of  alternatives  would  be  an  example  of  an  input  bias.  Also,  a 
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performance  evaluation  that  considered  investments  a  provider  made  in  achieving  outcomes 
would  be  an  example  of  an  input  bias.  In  our  paper,  we  investigate  the  susceptibility  of  decision 
makers  to  an  input  bias  when  evaluating  the  performance  of  logistic  service  providers. 

There  are  other  reasons  why  decision  makers  may  seek  out  component-level  process 
measures,  even  when  they  have  been  directed  to  look  for  system-level  outcome  measures. 
Process  measures  allow  a  better  degree  of  control  over  the  internal  workings  of  a  process. 

They  may  not  reduce  uncertainty  around  outcomes,  but  they  do  give  decision  makers  a  sense 
that  outcomes  are  more  directly  under  their  control.  Risk  preferences  vary  widely,  but  in 
addition  to  individual  differences  in  risk  aversion  or  risk-seeking  behavior,  decision  makers  tend 
to  prefer  controllable  to  uncontrollable  ones  even  to  the  extent  that  they  will  maintain  illusions 
about  the  degree  of  control  they  have  over  a  situation  (Langer,  1975).  The  preference  for 
controllable  risks  is  said  in  part  to  be  related  to  a  general  bias  decision  makers  have  that  their 
own  abilities  are  better  than  others’  (Howell,  1971).  Of  course,  part  of  the  logic  of  performance- 
based  outsourcing  is  that  providers  are  more  capable  of  dealing  with  the  internal  processes  of 
the  logistics  service.  But,  decision  makers  appear  to  maintain  this  preference  for  controllable 
risks,  and  to  support  their  bias  toward  exaggerated  self-assessments,  even  when  they  would  be 
better  off  with  less  control  (Klein  &  Kunda,  1994). 

In  delegating  the  decisions  on  how  to  accomplish  outcome  goals  to  a  provider,  programs 
seek  to  use  PBL  to  transfer  some  of  the  process  and  financial  risk  of  the  logistic  service  to  the 
provider;  in  contracting  to  deliver  outcomes  while  assuming  responsibility  for  processes, 
providers  accept  that  risk  at  a  specified  price.  The  assessment  of  these  risks  is  part  of  a 
business-case  analysis  required  for  every  implementation  of  PBL  in  the  Navy  (Young,  2003). 

To  our  knowledge,  however,  DoD-level  PBL  guidance  does  not  require  any  specific  measures  of 
outcome  risk,  or  process  risk  transfer. 

The  biases  and  heuristics  literature  make  it  clear  that  human  decision  makers  are  poor 
intuitive  statisticians  (Kahneman,  Slovic,  et  al.,  1982).  Indeed,  one  of  the  early  criticisms  of  that 
research  was  that,  in  part,  it  merely  represented  tests  of  intelligence  or  educational  achievement 
(Cohen,  1982).  As  Cohen  (1982)  pointed  out,  if  decision  makers  could  intuitively  grasp 
statistical  concepts,  what  would  be  the  point  of  offering  classes  about  these  concepts? 

However,  whether  it  is  a  question  of  education  or  irrationality,  it  seems  clear  that  most  decision 
makers  do  not  have  an  intuitive  model  that  allows  them  to  value  variance  in,  for  example, 
operational  availability.  In  our  paper,  we  investigate  the  tendency  for  decision  makers,  even 
when  trained  in  risk  assessment,  to  undervalue  the  impact  of  outcome  variance. 

The  investigations  in  our  paper  are  all  made  through  laboratory  experiments:  questionnaires 
asking  decision  makers  to  evaluate  PBL  scenarios.  The  results  have  only  limited 
generalizability  to  the  actual  management  of  extant  PBL  contracts,  or  to  the  valuation  and 
pricing  of  PBL  contracts.  However,  the  results  do  have  implications  for  the  continued  evolution 
of  PBL,  and  the  need  for  greater  specificity  in  guidance.  That  is,  if  decision  makers  under  PBL 
are  subject  to  the  same  limitations  as  decision  makers  in  our  study,  our  research  indicates  the 
need  for  the  DoD  to  develop  specific  guidance  with  regard  to  risk  measurement  and  valuation, 
and  to  require  comprehensive  system-level  models  to  value  and  price  component-level 
contracts,  and  evaluate  component-level  logistic-service  provider  performance. 
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